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LONG-TERM  GOALS 

The  long-tenn  goal  of  this  effort  is  to  improve  the  effectiveness  of  METOC  information  flow  to  the 
tactical  users.  This  information  flow  includes  both  the  data  flow  from  the  data  sources  to  the  METOC 
forecasters  and  the  environmental  impact  information  flow  to  the  warfighters.  As  part  of  the 
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information  flow,  the  project  intends  to  understand  how  forecasters  use  complex  visualizations  to  make 
a  forecast,  and  to  improve  their  workflow,  visualization  usage,  and  tools  to  increase  the  accuracy  of 
their  forecasts  and  decrease  the  time  spent  making  a  forecast.  To  achieve  this  goal,  key  components  of 
a  prototype  system  will  be  developed  to  test  the  hypotheses  and  refine  theories  of  the  information  flow. 
Ultimately,  the  prototype  of  the  information  flow  model  will  be  transitioned  to  the  METOC 
community  for  operational  use  and  application  in  the  warfare  areas.  This  information  interaction  model 
can  also  be  used  by  the  research,  development,  acquisition,  and  training  communities  to  identify  where 
significant  improvements  can  be  made  and  to  help  identify  metrics  to  evaluate  the  value  of  the 
improvements. 

OBJECTIVES 

The  scientific  and  technical  objective  of  this  project  is  to  improve  the  workflow  of  METOC  human- 
system  interaction  for  generating  useful  MEOC  products  to  warfighters.  In  this  project  a  key  objective 
is  to  develop  an  infonnation  interaction  model.  An  information  interaction  system  will  also  be 
prototyped  based  on  the  information  interaction  model  to  test  whether  a  more  efficient  method  for 
METOC  personnel  to  present  concise  tactical  weather  infonnation  to  the  warfighters  can  be  developed. 
We  will  focus  our  attention  on  the  METOC  information  flow  for  the  Air-Strike  warfighters. 

APPROACH 

A  team  of  cognitive  scientists,  METOC  researchers,  and  system  engineers  takes  the  iterative  observe- 
and-build-and-test  approach  to  the  problem.  We  apply  this  approach  to  a  series  of  experiments  where 
teams  of  forecasters,  working  in  a  simulated  or  actual  carrier  METOC  offices  (CV-METOC),  make  a 
pre-strike  mission  briefing  package.  We  observe  the  forecast  team  in  scenario-based  laboratory 
simulations  producing  the  weather  forecast  for  the  mock  strike  mission.  We  also  observe  ship 
forecasters  in  their  operational  environment  performing  their  duties.  We  use  cognitive  research  tools  to 
analyze  the  observations  during  the  experiment.  We  construct  the  information  flow  model  from  the 
analyses  of  the  observations,  and  we  test  our  hypotheses  in  the  following  experiments.  We  also 
develop  visualization  tools  for  assisting  the  information  flow  and  generating  tactical  products. 

The  initial  product  will  be  an  interactive  infonnation  model  which  will  depend  on  the  understanding  of 
not  only  the  METOC  forecasting  and  warfighter  needs,  but  also  the  cognitive  processes  involved  in 
forecasting  and  the  role  of  visualizations  in  these  cognitive  processes.  As  a  result  of  these  analyses,  we 
will  construct  two  models.  One  is  the  workflow  process  model,  which  reflects  the  individual  and  group 
behavior  in  developing  a  strike  forecast.  The  other  is  the  information  model,  which  reflects  the  utility 
of  the  end  products  for  specific  tactical  applications.  Finally,  the  interaction  and  feedback  processes 
within  the  workflow  and  infonnation  models  will  be  defined  and  evaluated.  Once  we  have  a  model 
that  can  be  described  and  tested,  we  can  build  a  prototype  that  implements  the  model.  This 
implementation  will  incorporate  processes  and  interactions  that  are  designed  to  improve  the  usefulness 
and  acceptance  of  the  forecast  products. 

WORK  COMPLETED 

This  research  project  is  a  collaborative  effort  among  several  research  groups  that  are  specialists  in 
cognitive  research,  meteorological  and  oceanographic  infonnation  systems,  and  strike  mission 
planning  research.  During  this  past  year,  we  have  analyzed  the  observations  collected  during  two 
experiments  conducted  in  July  2000.  The  analyzed  result  was  used  as  the  basis  of  our  hypotheses  for 
the  2001  Experiment,  which  was  conducted  in  Feb  2001. 
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2001  Experiment:  The  2001  Experiment  was  conducted  on  the  USS  CARL  VINSON  (CVN-70)  in  the 
OA  Division  office  (Weather  Office)  during  the  period  Feb  11-  16,  2001.  CARL  VINSON  was 
participating  in  one  of  the  pre-deployment  Battlegroup  workup  exercises  referred  to  as  Composite 
Training  Unit  Exercise  (COMPTUEX).  Observations  of  the  experiment  were  collected  during  the  first 
week  of  the  exercise  as  the  ship  operated  in  SOCAL  OP  AREA  (coastal  waters  adjacent  to  Southern 
California). 

The  purpose  of  the  experiment  is  to  test  the  following  hypotheses: 

•  Naval  tactical  operators  could  use  meteorology  and  oceanography  (METOC)  information 
that  is  more  accurate,  more  timely,  and  with  higher  resolution  than  is  being  routinely 
provided. 

•  Automation  of  the  repetitive  steps  in  the  forecast  process  could  help  standardize  procedures 
and  free  time  for  more  important  cognitive  tasks. 

•  Information-rich  intuitive  visualizations  could  help  forecasters  extract  more  knowledge 
from  the  weather  data  they  have  at  hand. 

•  The  technicians  who  support  the  forecasters  may  not  always  be  familiar  with  how  to 
retrieve  the  necessary  parameter  information  needed  as  inputs  for  tactical  decision  aids. 

From  the  observations,  we  evaluated  workflow  and  visualizations  used  by  METOC  personnel  to 
construct  weather  impact  information  in  support  of  naval  operations  in  general,  and  naval  air  strike  in 
particular.  The  observing  team  collected  forecasting  workflow  information  including  videotapes  of 
weather  office  procedures,  timings  of  each  task,  notes  of  forecaster’s  activities,  notes  of  weather  office 
personnel  interviews,  notes  of  pilot  interviews,  and  notes  of  staff  interviews. 

Specific  data  taken  from  the  OA  Division  to  assist  the  analysis  included  the  products  which  a  carrier 
weather  office  produces  during  tactical  operations,  the  data  required  to  create  these  weather  products, 
the  average  time  it  takes  to  create  these  products,  and  smaples  of  mesoscale  atmospheric  model  output 
(i.e.,  for  studying  small  scale  weather  features). 

The  operating  schedule  of  the  OA  Division  was  based  on  the  12-hour  watch.  Each  watch  consisted  of  a 
forecaster  (e.g.,  an  Aerographer  with  an  E6  grade  and  specialized  forecasting  training)  and  three 
technicians  (Aerographers  with  limited  METOC  training).  Daily  activities  of  the  experiment  can  be 
found  at  http://pluto.apl.washington.edu/metoc_hcs  .  The  forecaster  had  the  sole  responsibility  to 
produce  the  standard  set  of  the  products.  Technicians  were  on  call  to  perform  the  required  duties 
ranging  from  downloading  data/images  to  executing  tactical  decision  aids.  The  logistics  of  the  daytime 
observations  were  very  difficult.  The  observation  was  often  interrupted  by  the  OA  Division  daily 
duties  including  training  and  hosting  visitors.  The  physical  space  in  the  the  OA  Division  is  very 
limited,  in  many  occasions,  project  observers  had  to  leave  the  room  to  allow  the  OA  Division  to 
conduct  its  normal  business.  However,  during  the  night  shift,  observers  had  no  interruptions  to  record 
the  forecasting  procedures  (see  Figure  below). 
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Figure:  A  typical  time  analysis  summary  of forecaster’s  time  used  in  the  night  shift  forecasting. 
Overall,  forecasters  were  able  to  concentrate  on  the  task  at  hand  about  84%  of  time. 


Parallel  Experiment:  The  NUWC  team  member  had  an  excellent  opportunity  to  conduct  an  extended 
replication  of  the  METOC  experiment  at  Royal  Australian  Navy  (RAN).  This  parallel  experiment 
provided  us  an  opportunity  to  observe  a  totally  independent  group  performing  the  identical  function. 
The  data  collected  could  validate  our  Qualitative  Mental  Model  (QMM)  theory  on  how  METOC 
forecasters  formulate  their  thought  processes  and  make  their  forecasts.  Both  differences  and 
similarities  would  assist  us  in  understanding  the  workflow  process.  Expert  Interviews:  In  addition  to 
the  2001  Experiment,  the  team  made  visits  to  METOC  activities  to  obtain  their  approaches  toward  the 
METOC  data  workflow.  These  METOC  activities  included  Naval  Postgraduate  School,  Fleet 
Numerical  METOC  Center,  NPMOC  San  Diego  and  NPMOF  Whidbey  Island. 

RESULTS 

Validation  of  the  workflow  process  through  the  concept  of  the  Qualitative  Mental  Model  (QMM)  is 
most  significant.  Through  the  independent  data  collected  from  RAN,  the  concept  of  the  QMM  refined 
in  2000  is  still  valid. 

The  ship  communication  link  is  limited.  A  regional  METOC  center  can  assist  the  ship  forecasters  to 
gather  information  by  staging  “critical”  data  at  one  central  location.  From  the  data  collected  in  the 
2001  experiment,  the  team  has  a  list  of  products  that  the  METOC  regional  sites  should  be  gathering. 
Currently,  the  team  members  are  working  with  Naval  Postgraduate  School  to  refine  this  “critical”  data 
list.  In  addition,  the  tactical  weather  impact  infonnation  should  also  be  staged  by  the  regional  center. 
As  we  observed  that  the  METOC  data/infonnation  have  gone  to  the  warfighters  directly  in  some 
circumstances,  it  is  our  conclusion  that  these  tactical  impact  products  can  be  automatically  ingested 
into  the  tactical  impact  spreadsheet  used  by  the  Battlegroup. 

Currently,  the  forecasters  are  taking  a  default  approach  in  gathering  the  information  (use  whatever  is 
available),  instead  of  the  funneling  approach  in  which  one  would  systematically  seek  out  appropriate 
information.  In  many  occasions,  the  team  observed  that  the  forecasters  use  the  mesoscale  model 
outputs  without  clear  understanding  of  their  strengths  and  weaknesses  as  compared  to  use  of  a  global 
model  (i.e.,  the  mesoscale  and  global  model  output  are  used  interchangeably).  A  forecast  checklist 
would  be  very  helpful  for  inexperienced  forecasters  to  take  a  systematic  approach  to  forecasting. 
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Much  forecast  information  produced  by  the  forecasters  is  repeated  in  many  different  forecast  products. 
The  forecast  office  can  use  an  information  storage  center  and  message  generation  application  for 
transferring  the  information  from  one  format  to  the  other.  The  benefit  of  this  infonnation  management 
is  to  force  consistency  in  the  product  content. 

IMPACT/APPLICATIONS 

This  important  collaboration  brings  together  researchers  from  many  disciplines  to  study  the  critical 
problem  of  information  flow  from  METOC  to  Strike  planners  by  focusing  on  the  nature  of  workflow. 
In  the  2001  Experiment,  research  has  progressed  through  carefully  designed  steps.  We  have  advanced 
our  theory  on  the  METOC  workflow,  and  we  have  fitted  our  observation  methodology  for  use  on 
board  an  aircraft  carrier.  The  objectives  of  the  2001  Experiment  on  board  the  USS  CARL  VINSON 
were  to  validate  workflow  concept  through  the  QMM  and  to  test  if  forecasters  can  get  tactical  weather 
information  from  the  mesoscale  weather  prediction  model  output.  We  believe  that  the  workflow  model 
will  be  the  key  for  building  the  future  METOC  information  infrastructure.  From  the  2001  experiment 
results,  we  suggest  that  1)  data/infonnation  staging  for  the  forecasters  is  a  key  area  for  improvement  of 
the  workflow,  2)  the  data  staging  can  assist  forecasters  to  expedite  the  “Verily  and  Adjust”  steps  in  the 
QMM,  3)  the  automatic  linkage  of  tactical  impact  variables  can  save  forecasters  valuable  time  in 
producing  a  product,  and  4)  the  in-field  observations  reveal  more  than  the  laboratory  experiment.  The 
scenario  based  laboratory  experiment  is  an  excellent  methodology  for  discovering  workflow  problems, 
but  actual  observation  of  the  ship  operations  cannot  be  neglected. 

Some  of  these  results  will  be  presented  to  the  Naval  METOC  Personnel  Training  Center  in  Gulfport, 
Mississippi  and  can  impact  their  mid-career  level  training  for  METOC  personnel.  We  believe  that  our 
analyses  show  the  infonnation  interaction  that  leads  to  improved  forecasting  in  support  of  strike 
operations.  Our  analyses  can  also  be  used  for  identifying  common  errors  and  areas  where  improved 
training  would  be  helpful. 

TRANSITION 

The  findings  of  this  study  will  fonn  the  basis  for  METOC  data  workflow  design.  The 
recommendations  are  transitioned  to  the  6.3  Environmental  Visualization  project.  The  approaches  of 
the  FNC  Environmental  Visualization  (EVIS)  project  (starting  in  FY-02)  are  based  on  the  project 
findings  as  summarized  in  the  Results  Summary. 

The  findings  of  this  study  will  also  be  transitioned  to  the  Naval  METOC  Personnel  Training  Center  for 
developing  their  training  curriculum. 

The  workflow  observation  technology  can  be  transitioned  to  the  METOC  community  for  discovery  of 
workflow  problems  in  other  disciplinary  areas. 

RELATED  PROJECTS 

Environmental  Visualization.  FNC/KSA  ONR  $1.2M,  FY-2002  (Balias) 

Nowcasting  for  Next  Generation  of  Navy,  ONR  $400K/yr,  1999-2004  (Cook) 

Human  Centered  Scientific  Visualization.  ONR/NRL,  2000-2002,  (Trafton) 
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Integration  and  Visualization  of  Multi-Source  Infonnation  for  Mesoscale  Meteorology:  statistical  and 
Cognitive  Approaches  to  Visualizing  Uncertainty.  ONR  MURI,  $5M,  10/2001  -  4/2006  (Raftery) 

SUMMARY 

METOC  forecasters  not  only  extract  information  from  visualizations,  they  create  a  qualitative  mental 
model  (QMM)  and  reason  with  it.  They  then  use  their  QMM  to  generate  quantitative  information.  We 
have  refined  this  model  this  year  to  explore  how  experts  and  novices  differ  in  their  use  of  their  QMM 
and  the  forecasting  process  in  general,  as  well  as  replicating  the  basic  finding. 

Through  the  2001  experiment,  we  found  that  a  data  staging  strategy  can  be  used  to  assist  the  ship 
forecasters  gathering  information.  The  data  and  information  can  be  prepared  and  staged  to  improve  the 
workflow  around  the  ship  forecasters.  Concrete  steps  have  been  adopted  in  the  work  plan  for  the  new 
ONR  Future  Naval  Capabilities  (FNC)  project. 
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